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ELEMENT PERSISTENT IDENTIFICATION 

TECHNICAL FIELD 

The technical field relates to graphical user interface components of a computer 
5 program. More particularly, the field relates to computer implemented methods and 
systems for identifying elements or components of a graphical user interface system of 
a computer program. 

BACKGROUND 

A typical mechanism for implementing user interaction with a computer may 

10 involve the use of a graphical user interface (GUI) associated with a computer program 
running on the computer. Generally speaking, user interface elements (hereafter, UI 
elements) are aspects of a computer system or program which may be seen (or heard or 
otherwise perceived or interacted with) by an end user and used to control the operation 
of a computer system or program through commands and other mechanisms. A 

15 graphical user interface emphasizes the use of graphical UI elements which are 

responsive to user input (e.g., mouse events, keyboard data entry etc.) for interacting 
with a computer system or program. Common UI elements familiar to most users 
include, buttons (e.g., "OK" or "Cancel), edit boxes, list boxes, combo boxes, scroll 
bars, pick lists, pushbuttons, radio buttons and components of World Wide Web pages 

20 (e.g., hyper-links, documents and images). 

In a typical computer program it is not uncommon to encounter literally 
thousands of such UI elements. Also, some of the same UI elements may appear 
multiple times as part of different composite UI elements depending on a current 
context of the program's use. Also, the same UI elements may appear in other 

25 computer programs as well. Nevertheless, each instance of a UI element may need to 
be uniquely identified for various reasons such as programming the functionality of the 
program hosting such UI elements, testing such a program or for use in otherwise 
identifying the particular UI element to another program module in communication with 
the host program. For instance. Assistive Technology (AT) software, like screen 



SYM/sym 3382-66149 10/23/03 EXPRESS MAIL LABEL NO.: EV352377952 

DATE OF DEPOSIT: October 23. 2003 

readers, screen enlargers or alternative input devices, as well as test automation 
programs that communicate with application programs may need to clearly identify UI 
elements hosted within the appUcation programs. More particularly, in one potential 
scenario, suppose an application program tester has found a defect related to a UI 
element in an application program and wants to continue to exercise the application at 
the same UI element to determine whether the defect was an aberration or permanent. 
In order to do so, the tester may want to reboot his computer and get back to the same 
defective UI element for testing. 

However, currently there is no reliable mechanism of identifying these UI 
elements across reboots, across instances of an application, across localized versions, 
and across builds. Currently, depending on the operating system platform, it is possible 
to use a combination of role, name, location, position among siblings (e.g., 4^^ child 
among n children), and other things. However, none of these properties on their own 
are sufficient; and many of them are fragile (e.g., location, name). 

Thus, there is a need for systems and methods to provide identifiers for a UI 
element that persist across instances of an application, across localized versions, across 
builds, across reboots and other similar events. More particularly, there is a further 
need for platform level Application Program Interfaces (APIs) that can be used by a 
client program to receive persistent identifiers related to a source program and use such 
an identifier to locate a particular UI element in a target program. 

SUMMARY 

Described herein are Application Programming Interfaces that enable client 
applications to request and receive identifiers for user interface elements in a graphical 
user interface of a computer program, wherein the identifiers are capable of persistently 
identifying the user interface elements across different instances of the computer 
program, across different builds of the program, across reboots of the computer running 
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the computer program etc. A persistent identifier is one that is not fi-agile and maintains 
its identity across events such those mentioned above. 

In one aspect, the persistent identifiers are generated by using a hierarchical tree 
structure representation of the graphical user interface comprising the user interface 
5 element of interest. The identifiers are made persistent by including in the identifier, 
identifying information not only of the user interface of interest but also as much of the 
identifying information as is appropriate of the parent elements above the user interface 
of interest in the hierarchy of the tree structure. Such information describes a 
hierarchical path of inheritance for the user interface of interest within the graphical 

10 user interface. The path may be referred to as an element path and the identifier 

comprising a description of the path may be referred to as an element path identifier. 

Furthermore, a set of Application Programming Interfaces are described herein 
that enable a client program to locate a target user interface in a graphical user interface 
of a target computer program by using the element path identifier of the target element. 

15 For example, by using APIs for generating persistent identifiers a client program can 
mark a user interface of interest and later use the persistent identifier to locate the user 
interface of interest in another instance of the program, across another build of the 
program and across other events that would have rendered any of the fragile identifiers 
useless. 

20 In another aspect, a tree structure representation of a graphical user interface 

may comprise at least one branch portion that is associated with a strong name and has a 
scope inside of which elements of the strongly named branch are guaranteed to be 
uniquely identifiable by using a named branch element identifier. Thus in one 
embodiment of generating an element path identifier, elements of a named branch 

25 portion may be uniquely identified by their named branch element identifier alone. In 
yet another embodiment, sections of an element path comprising strongly named 
branches may be identified by the strong name of the named branch and the named 
branch element identifier of the lowest element still within the scope of the named 
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branch portion. This reduces the quantity of identifying information stored in a elment 
path identifier data structure. However, the efficiency of searches for target user 
interface elements can be improved by adding information related to all the elements in 
named branch portion. 

The persistent identifier can be further improved by adding additional 
information within an element path identifier for reducing any residual ambiguity. For 
instance, sibling order information can focus a search and reduce ambiguity. 

Additional features and advantages of the systems and methods described herein 
will be made apparent from the following detailed description that proceeds with 
reference to the accompanying drawings. 

BRIEF DESCRIPTION OF DRAWINGS 

FIG. 1 is a block diagram illustrating a tree structure representation of a 
graphical user interface with multiple user interface elements arranged in a hierarchy. 

FIG. 2 is a block diagram illustrating the tree structure representation of FIG. 1 
with an element path that may be used to persistently identify one of the multiple user 
interface elements. 

FIG. 3 is a flow chart describing an overall method of generating or recording an 
element path identifier of a selected source user interface element. 

FIG. 4 is a flow chart describing an overall method of searching and locating a 
target user interface element according to its element path identifier. 

FIG. 5 is a flow chart describing in detail one method for generating an element 
path identifier for a source user interface element in a tree structure representation by 
starting at the source element leaf node and recording exposed identifier information of 
selected parent element nodes above in the hierarchy. 

FIG. 6 is a flow chart describing in detail one method of locating a target user 
interface element in a tree structure representation of a user interface based on its 
element path identifier starting at a designated element path root node and successively 
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pruning the tree by searching for the target element in selected branches according to 
the element path identifier. 

FIG. 7 is a block diagram of a system for generating element path identifiers of 
selected user interface elements and locating user interface elements based on their 
5 element path identifiers. 

FIG. 8 is a block diagram illustrating another element path of a source user 
interface element used for generating its persistent identifier. 

FIG. 9A is a block diagram of an element path identifier data structure for the 
exemplary element path of FIG. 8. 
10 FIG. 9B is a block diagram of exemplary fields or members of the element path 

identifier data structure of FIG. 9 A. 

FIG. 10 is a block diagram of a branch portion of a user interface tree structure 
representation (e.g., FIG. 8), wherein one of its sub-branches has a strongly named root. 

FIG. 11 is a block diagram of fields or members of an element path data 
15 structure (e.g., FIG. 9A) with at least one field designated for a named branch element 
identifier. 

FIG. 12 is a block diagram illustrating an element path of a user interface 
element of interest within a named branch portion of a user interface tree structure. 

FIG. 13 is a block diagram illustrating an element path of a user interface 
20 element of interest within a named branch portion of a user interface tree structure, 
wherein the path does not include all of the parent elements of the user interface 
element of interest. 

FIG. 14 is a block diagram illustrating a user interface element of interest which 
has been moved fi-om its original location recorded in its element path identifier. 
25 FIG. 15 is a block diagram illustrating an element path for persistently 

identifying a user interface of interest that may be moved fi-om its original location, 
wherein the path includes the identifying information related to its immediate parents in 
its original location. 
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FIG. 16 is a block diagram of a tree structure representation of a user interface 
wherein the various elements of the user interface may be created within different user 
interface platform types. 

5 DETAILED DESCRIPTION 

An exemplary hierarchical representation of a graphical user interface 

Although individual elements of a graphical user interface (e.g., a radio button) 
10 may appear to the user as a single composite item, it may actually be represented in the 
computer as a number of separate items or sub-elements that have been combined 
together. Furthermore, each of these sub-elements themselves can be composed from 
other sub-elements. In this manner, UI elements can serve as building blocks for 
building other, more complex, UI elements. Such an approach is useful because the 
15 software managing the user interface (e.g., the user interface framework) can re-use the 
definitions of certain common elements when assembling them into composite 
elements. 

FIG. 1 illustrates a tree structure representation 100 of a typical graphical user 
interface (hereafter, GUI) on a computer. More particularly, FIG. 1 illustrates how 

20 elements of a GUI can be shown as nested within each other in order to accurately 

describe their behavior (e.g., visual or fiinctional). At very top of the tree structure 100 
may be a desktop element 101 that is representative of the computer's desktop. The 
desktop element 101 may have represented within it several application elements (e.g., 
105 A, 105B and 105C) that have been invoked and ready to execute according to a 

25 user's instructions (e.g., a typical Microsoft® Windows desktop may have several 

instances of applications such as Microsoft® Word, Microsoft® Excel®' etc. loaded and 
ready to be executed). At a lower level in the tree structure hierarchy may be several 
frames (e.g., 1 lOA, HOB and 1 IOC) that may be associated with an application 105B 
(e.g., a word processor appUcation may have several frames visible to a user at any 
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given time). Within each of the frames (e.g., HOB) may be several documents (e.g., 
1 15A and 1 15B) and each document may contain within it several control UI elements 
120A, 120B and 120C (buttons, listboxes, etc.). Control UI elements (e.g., 120A, 120B 
and 120C) may themselves be composites of other UI elements. For example, a dialog 
5 box, or a combo box, in turn may contain other control UI elements such as a button 
control at 125. Furthermore, the button element 125 itself may comprise other control 
UI elements such as 130 and such nesting can go on further deeper depending on the 
user interface and its component elements. 

10 Exemplary identifiers for elements of a graphical user interface 

Any one of the elements of the user interface illustrated in FIG. 1 may be 
associated with a nimiber of different identifiers, which can be used to distinguish that 
particular UI element from other UI elements. Depending on the type of operating 

15 system platforms each instance of a UI element may be assigned at least one identifier. 
For example, in Microsoft® Windows based operating systems, applications may be 
associated with module identifiers that uniquely identify applications within a given 
desktop context. Also, many user interface platforms (e.g., Microsoft® Windows, 
Swing® for Java) allow for an alphanumeric identifier for UI elements commonly 

20 referred to as control IDs. Furthermore, in an objected-oriented environment, UI 

elements may also be associated with a class name associated with the control class to 
which they may belong. For instance, in a Microsoft® Windows based system, common 
UI elements such as combo box, list box, and button are associated with class names 
such as ComboBox class, ListBox class, and Button class respectively. Similarly, other 

25 user interface frameworks may have names for their respective classes of UI elements. 

Although it is possible to identify UI elements using identifiers such as those 
listed above, none of them can singularly provide a strong identifier that is likely to 
persist across a reboot of the computer running the program, across a different build of 
the program when still in development, across the opening of an another instance of the 
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same program, or opening of the same type of program on another computer. In such 
situations these identifiers alone prove to be fragile or too ambiguous. For instance, 
control identifiers (control IDs in Microsoft® Windows) by themselves are not very 
unique. On a given Microsoft® Windows desktop, for instance, two different UI 
elements (e.g., a button and a dialog box) running on two different applications may be 
assigned the very same control ED. Thus, in this example, searching by a control ID 
alone cannot unambiguously identify a selected one of those UI elements. 

However, combining the control ID with another identifier may be helpfiil. 
Take for instance the same example as above, even if the control ID for the button 
element and the dialog element were same, if a composite identifier combined the 
control ID and the control class name associated with the individual UI element then the 
button UI element and the dialog box UI element may distinguishable fi-om each other. 
Similarly, generating composite identifiers having additional information capable of 
identifying a UI element can increase the strength of the identifier's uniqueness and 
reduce the possibility of ambiguity. 

An exemplary UI element path description as a persistent identifler of UI elements 

Adding as much identification information that is directly associated with a UI 
20 element into a composite identifier of a UI element may be helpful in identifying it 
unambiguously. However, given the countless number of UI elements and the 
complexities involved, it is may not succeed all the time. Thus, yet another 
improvement may be to also add identification information related to parent elements to 
the composite identifier of a UI element. For instance, a UI element path description 
25 identifier can be provided which comprises not only identification information related 
to a target UI element for which a unique persistent identifier is to be generated but it 
may also include information related to its parent elements and possibly even its 
children and sibling elements. FIG. 2 is an illustration of the overall concept of an 
element path identifier for a target UI element 205 that may be part of a user interface 



10 
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represented as a tree structure 200. As shown in FIG. 2, an element path identifier may 
be generated for a target UI element 205 that comprises a collection of identifying 
information related to parent UI elements (e.g., 215A- 215F) on an element path 
(e.g. ,2 10) that describes the hierarchical arrangement between a target leaf UI element 
5 (e.g., 205) and a root element at desktop 215A. Elements (e.g., 216) not forming the 
path 210 are shown in FIG. 2 with broken border lines for the purpose of distinguishing 
them from elements that are within the path (e.g., 215D). 

An element path description identifier may be able to serve as a strong identifier 
for a UI element because, even if the target UI element has other identifiers (e.g., 
10 module name, control ID, control class name etc.) that are identical to another UI 

element, it is less hkely that they both would have the same parent elements, also with 
identical identifying information. Through concept of a path related identifier a UI 
element's unique hierarchy and parentage can be leveraged to identify it uniquely and 
persistently. 

15 

Exemplary methods of recording or generating identifiers of UI elements 

Persistent identifying information related to a selected source UI element may 
be recorded and later used as a parameter for a method of searching and locating a 

20 target UI element that matches the identifier of the source UI element. For example, the 
source element and target element may be different instances of the same element. For 
instance, a button of interest in a word processor program may be given a persistent 
identifier and the identifier may be used to locate the same button on the same program 
being run on a different computer. As shown in FIG. 3, at 310, a specification of a 

25 source UI element for which an identifier is to be generated may be received. Upon 
receiving such a specification, at 320, an element path identifier may be generated for 
the specified source UI element that provides a persistently unique identification of the 
source UI element. 
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According to one embodiment, the method of recording or generating an 
element path description identifier for a source UI element may be implemented in a 
software component as described in FIG. 5. At 510, the method of recordation 510 can 
start at a leaf node representative of the source UI element (e.g., 205) within a 
5 hierarchical tree structure representation (e.g., 200) of the user interface of a source 
program within which the source UI element itself is a hosted component. At 520, any 
available identifying information related to the source UI element 205 may be recorded 
as a field in a data structure related to the element path identifier. Then at 530, the 
identifying information related to the next parent node (e.g., 215F) of the source UI 

10 element (e.g., 205) is recorded. At 540, this process is continued until a parent node 
designated as an element path's root node (e.g., the desktop node 215A or the 
application node 215B) is encountered, upon recording which the process of recordation 
may be ended at 550. 

The information recorded at each UI element in an element path (e.g., 210) may 

15 be a function of what is exposed in that particular UI element's object model and may 
also depend on the user interface platform used to create the UI element. For example, 
it is possible for a user interface to comprise UI elements from a number of different UI 
platforms (e.g., Win 32 by Microsoft® Corporation, and Swing for Java etc.). Thus, the 
method of recordation should be capable of identifying a UI element as belonging to a 

20 certain user interface platform and record its identifying information accordingly. The 
actual information about each UI element recorded may include such items as control 
ED, module identifier, control class name etc. Initially, as the method is progressing up 
a tree representation, some identifying information (e.g., module identifier for 
identifying a UI element 205 as belonging within a certain application module 215B) 

25 may not be exposed until a higher element is reached. In that event, such information is 
recorded upon being exposed. 

In the previous example, the topmost node in the user interface hierarchy 21 5 A 
was also designated as the element path root node where the recording of the identifier 
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information for the element path identifier was to be stopped. However, it is possible to 
designate a lower element (e.g., the application node 215B or the frame node 215C) to 
be designated as the element path root. This may be useful where a particular client 
system can ensure strong and persistent identification of elements up to a certain level 
5 in the user interface hierarchy or can devise other local methods of searching and 
identification of UI elements at certain levels. The element path root node may be 
provided as a parameter to the process 510. 

Exemplary methods of searching UI elements according to their identifiers 

10 

Once an element path identifier is recorded, for instance, it may be used as a 
parameter in a search method applied to a target application to locate a UI element 
matching the identifier within the application. Such a process may be implemented in a 
software component for searching UI elements as described in FIG. 4. At 410, a 

15 specification of an element path identifier associated with a target UI element may be 
received. Upon receiving a specification of the element path identifier, at 420, the 
target UI element described by the identifier may be located. 

According to one embodiment, the method 420 of locating the target UI element 
upon receiving a specification of an element path identifier may be implemented in a 

20 software component as described in FIG. 6. At 610, the search process may begin at a 
designated element path root node (e.g., 215A or a 215B) in a tree structure 
representation (e.g., 200) of a user interface of a target application (i.e., a computer 
program that may be hosting the target UI element). The designated root node of the 
element path can be any node within the tree structure; it does not have to be the 

25 topmost node such as the desktop node 21 5 A. Generally, a search process is unlikely to 
begin at a desktop node 215A, unless, for example, the search scenario involves a 
system having multiple desktops (e.g., involving multiple operating system platforms). 
Then at 620, fields of the element path identifier data structure are evaluated to 
determine if any of the identifier data described therein matches the identifier data 
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exposed at the root node (e.g., 215A or 215B). If no such match is determined, then at 
621, the search process is ended. However if a match is in fact determined, then at 630, 
the search is advanced to other levels below in the hierarchy of the tree structure to 
determine which of the sibling elements at the next level exposes identifiers that match 
5 a corresponding field of the element path identifier. If no such match is determined, 
then at 63 1 , the search may be suspended. Thus, the search may be progressed below to 
even lower levels in the hierarchy until at 640 one of the child nodes exposes identifiers 
that match a corresponding identifier associated with a target UI element in its element 
path identifier. If a child node corresponding to the target UI element is not determined 

10 at 640 then the search is retumed to step 630 to continue the search until such a child 
node is found. Once a child node corresponding to the target UI element is found then 
the search is ended at 65 1 . 

The search process 600 described above prunes the tree structure 200 by 
progressing along the element path 210 one level at a time wherein one of the sibling 

15 elements forming a separate branch is distinguishable over the others of its siblings. 
Such a process eliminates the need to search every branch in a tree structure. However, 
it is also possible to search all branches from a certain branch root of the tree structure if 
identifying information that distinguishes a branch root node firom its siblings is not 
available in the element path information. For example, the branch root node 215D 

20 may not be distinguishable from the branch root node 216 based on the element path 
information. However, such a search process may be more complex and would 
certainly be more time and resource consuming than the process described with 
reference to FIG. 6. 

25 An exemplary system for generating and using element path identifiers 

The methods of generating element path identifiers for selected source UI 
elements and the complementary process of identifying or locating a target UI element 
according to such an identifier may be implemented within a system as shown in FIG. 



SYM/sym 3382-66149 10/23/03 



EXPRESS MAIL LABEL NO.: EV352377952 
DATE OF DEPOSIT: October 23. 2003 



-13- 

7. The system may comprise an element path engine 710 programmed for generating 
element path identifiers of source UI elements hosted within a source program 730 upon 
receiving a request to do so via a client program 720. Furthermore, the element path 
engine 710 may also locate or identify a target UI element within a target program 715 
5 based on an element path identifier provided via a client program 720. The 

communication between client program 720 and the element path engine 710 may be 
enabled via an Application Programming Interface (API) 725 which can allow the client 
program to pass function calls including element path description identifiers as its 
parameters to fiinctions implemented for searching of target UI elements. Also, the API 

10 725 may be fiirther capable of passing function calls including specification of one or 
more source UI elements as parameters to functions implemented for generating 
element path identifiers of specified UI elements. Embodiments of such APIs will be 
described in further detail below. 

The client program 720 may be a test program which is exercising an instance of 

1 5 a program (e.g., 730) such that the test program may identify a source UI element of 
interest for which it may request a persistent identifier such as its element path identifier 
through the element path engine 710. Later on, the test program (e.g., the client 720) 
may want to use the persistent identifier to locate another instance of the same UI 
element in another program (e.g., the target program 715). The source program 730 and 

20 the target program 715 may be located on the same or different computers. 

In yet another scenario, the client program 720 may use the element path engine 
710 to record a user's access of various UI elements within a source program 730 by 
identifying the accessed UI elements using the persistent identifiers and later play back 
the user's access scenario on another program (e.g., 715). For example, such an 

25 implementation may be helpful in creating a help menu where users are walked through 
the procedures of using a program by the way of an example. The above examples are 
meant to be illustrative and other uses for a persistent identification of UI elements are 
possible. 
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An exemplary element path description data structure 



FIG. 8 illustrates an exemplary tree structure representation 800 that is 
representative of a user interface. The UI element of interest labeled *7' 805 may be 
5 persistently identifiable via an element path description that comprises identifier 
information related to the UI element of interest 805 as well as its direct and indirect 
parent elements. Thus, the combination of parent elements '\\ '2\ '3\ '4\ '5\ and '6' 
(801-803) may form an element path 815 which can be used to provide a persistent 
identifier for the UI element of interest 805. 

10 FIG. 9 A illustrates an exemplary data structure for storing element path 

identifier information corresponding to the element path 805 of FIG. 8. An element 
path description identifier 900 may comprise nodes *l-7' 910 correspondingly 
associated with identifiers 'A-G' 920. As shown in FIG. 9B, each of the identifiers 'A- 
G' 920 may comprise fields such as control ID 921, control class name 922, module 

15 name 923 etc., as described above. Other identifiers as appropriate may be provided 
that can fiirther help distinguish each UI element over other UI elements. As noted 
above, one factor determinative of what information may be exposed at each node of a 
tree structure representation of a user interface may be the user interface platform used 
to create the individual UI element. 

20 

An identifier for a UI element guaranteed to be unique within a scope 

As noted above, identifiers of UI elements within an element path can comprise 
a wide range in the strength of their identity with a particular UI element. Thus, a 
25 process of recording or generating an element path or even the process of searching for 
a UI element according to its element path can be adjusted depending on the strength of 
the identifier. For instance, suppose the branch representation 1010 of FIG. 10 
represents a branch portion of the tree structure representation of 800 of FIG. 8, Within 
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the branch portion 1010 may be a another branch portion 1020 with a named branch 
root 1021 such that within a given scope of the branch (e.g., between the named branch 
root 1021 and the child leaf node 1025) all the children nodes (e.g., 1022, 1023, 1024 
and 1025) are uniquely identifiable from each other by an identifier. In this example, 
5 that would mean that within the given branch scope (e.g., between 1021 and 1025) each 
of the children nodes (e.g., 1022, 1023, 1024 and 1025) may not have any ambiguity 
between each other. Also, even if the tree structure is changed by moving one of the 
children elements (e.g., 1022, 1023, 1024 and 1025) to another location within the 
scope the moved child element would retain its unique identifier. For example, such an 

10 identifier is possible when a branch portion 1020 is related to a composite UI element 
that is created and stored under a specially strong name and all the component elements 
can be uniquely identified within the scope of the branch portion 1020. This may be 
possible in those instances when a user interface represented by the tree 800 comprises 
components that are themselves composite UI elements (e.g., 1020) that were created 

1 5 and stored under a strong name and later associated with the larger user interface 800. 

An exemplary element path identifier of a UI element in a user interface 
tree representation comprising strongly named branches 

20 The strong name associated with a named branch 1020 may be a file name or 

any other type of strong identifier of the named branch root 1021 . For example, such a 
name may be referred to as a named branch root identifier and each UI element within a 
scope related to the named branch root may be uniquely identifiable by a named branch 
element identifier (hereafter, named branch element ID). An element path identifier 

25 data structure for a user interface comprising such a named branch may need additional 
fields for identifying UI elements (e.g., 1022) that belong within the scope of named 
branch 1 020. For instance, FIG. 1 1 illustrates a data structure 1 1 00 for an element path 
description identifier including a named branch element ID field 1110. The named 
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branch element ID field 1 1 10 is capable of uniquely identifying a related UI element 
within a scope associated with a respective named branch. 



Methods of recording element path identifiers of UI elements and searching 
5 for UI elements by such identifiers in a user interface tree structure with strongly 

named branches 

The user interface tree structure representation 1010 comprising a named branch 
portion 1020 may have an element path corresponding to a UI element 1025 as shown 

10 in FIG. 12. The element path 1215 for the UI element 1225 is shown comprising 
identifier information including information related to its parent UI elements 1205, 
1221, 1222, and 1224. The information of UI elements not in this path (e.g., 1223) are 
not included. Using the process 500 described above an element path description 
identifier may be recorded including all parent elements between a designated element 

15 path root node and a source UI element. However, as noted above with reference to 

FIG. 10 certain UI elements (1022-1025) may be guaranteed to be uniquely identifiable 
within a given scope and in that event, there may not be a need to record (in an element 
path identifier) identity information regarding all of the parent elements of UI element 
within such a scope. 

20 Thus, as shown in FIG. 13, the parent elements 1322 and 1324 of the UI element 

of interest 1325 are not peurt of the element path 1315 and no information regarding their 
identity or their special relationship with the UI element of interest 1325 is recorded as 
part of its element path identifier. This is possible because within the scope of the 
named branch portion 1320 the UI element is guaranteed to be uniquely identifiable by 

25 its named branch element ID (e.g., 1110). By simply recording a named branch element 
ID 1 1 10 a named branch element may be located within its uniqueness scope by 
systematically searching all sub-branches of the named branch portion 1320. However, 
at 1305 identification information for a parent node that is outside the scope of the 
named branch 1320 may still need to be recorded. Altematively, the parent node 1305 

30 or even one of its parent nodes above in the hierarchy may be itself a part of another 
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named branch within which scope it is uniquely identifiable. In that event, only their 
corresponding named branch element ID may be recorded as part of an element path 
identifier. 

Strictly recording only the minimum amount of identification information that is 
5 necessary for locating a UI element may not always be helpfiil for increasing the speed 
of a search. The named branch 1320 comprises relatively few sub-branches (two) and a 
small number of sub-elements. This may not always be true. Searching for a sub- 
element by its element path identifiers which do not contain the identification 
information for all its parent elements may mean all branches have to be exhaustively 

10 searched which can be time and resource consuming. Thus, including (in an element 
path identifier) the identification information of parent elements of a named branch UI 
element (as shown in FIG. 12) can improve the speed of searching for that UI element. 

This can also be helpful in other circumstances. For instance, FIG. 14 illustrates 
a scenario wherein the named branch portion 1420 is shown with the UI element of 

15 interest 1325 which was originally shown in FIG. 13 as a child of the element 1324. 
However, perhaps in a later build of this user interface, the UI element of interest 1325 
has been moved to be the child of 1 323 UI element. If the element path identifier for 
the UI element of interest 1325 recorded a named branch element ID for the UI element 
of interest 1325 then it can be located in a playback scenario even if it has been moved 

20 to another sub-branch 1425 of the named tree branch 1420. This may be possible 
regardless of whether the identifier information of its original parents 1324 and 1322 
were recorded or not. 

However again in this scenario, as before, recording the identifier information 
regarding parent elements 1322 and 1324 of a UI element of interest 1325 may help 

25 improve the efficiency and speed with which it is located later. For instance, as shown 
in FIG. 15, a search method may first direct the search for the UI element of interest 
1325 in a branch 1510 where according to a previously recorded element path 
description identifier the UI element of interest 1325 was to be located. However, only 
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in the event the UI element is not located in the expected location then the search may 
be directed back to the nearest named branch root 1320 and the rest of the sub-branches 
(e.g,, 1520) of the named branch 1420 may need to be exhaustively searched to locate 
the UI element of interest 1325. In this scenario, including the identification 
5 information for the parent elements 1322 and 1324 may be optional. However, such 
information may help make the search process more efficient. 

Methods of recording element path identifiers and searching by such 
identifiers in a user interface tree structure with strongly named branches nested 
10 within other branch types 

FIG. 16 illustrates a tree structure representation 1600 of a user interface 
wherein several elements of the interface are created using several different user 
interface platforms with differences in identifier information. For instance, branch 

15 portion 1610 may be created within a first user interface platform type Platform 1. 
Branch portion 1620 may be created using a second user interface platform type 
Platform 2, which is capable of providing strongly named branches with a certain scope 
within which each element is uniquely identifiable. Then at yet another level, elements 
of the user interface representation may be created using yet another user interface 

20 platform type Platform 3. Also, UI elements created using different user interface 
platforms may be nested within each other. 

Furthermore, the process of recording an element path description and the 
process of searching for target UI elements can adaptively record information based on 
the platform affinity of the UI elements on a given path. 

25 

Embodiments of APIs for implementing methods of recording element path 
identifiers of source UI elements and searching for target UI element in a user 
interface representation using such an identifier 

30 The method of recording an element path identifier for a source UI element 

(e.g., the methods of FIGS 3 and 5) may be implemented using API software 
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components (e.g., 725) including function calls that client programs 720 can call on the 
element path engine 710. The same may apply to methods of searching for a target UI 
element (e.g., FIGS. 4 and 6) based on its element path identifier (e.g., FIGS. 9B and 
11) 

5 The API for generating or recording an element path identifier for a source UI 

element may be as follows in code representation: 

Pubhc: static ElementPath* GetElementPath (LogicalElement *logical element); 

10 The above GetElementPath method receives an input parameter "LogicalElement" and 
retums as its result an ElementPath. The LogicalElement parameter can be any local 
identifier (e.g., Control ID) of a source UI element for which an element path 
description is to be generated. Such an identifier need not be a persistent identifier but 
it should at least provide a pointer to a starting point at which recording of an element 

1 5 path description may begin. The result returned is the ElementPath which can be a data 
structure implemented as shown and described with reference to FIGs. 9B and 1 1 . 

The API for generating an element path description identifier for a source UI 
element may also be implemented as follows in code representation: 

20 Public: static String* GetElementPathString(LogicalElement * logical element); 

In this implementation the result returned is not an ElementPath data type instead it is a 
string type description of an element path. This may be useful for cutting and pasting 
an element path description into pieces of code. For instance, a test case code can use 
25 string descriptions of a UI element's element path identifier to allow a test program 
locate and exercise the UI element. 
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Complementing the above two APIs may be APIs for searching or locating 
target UI elements based on their respective element path descriptions. Such APIs may 
be as follows: 

Public: static LogicalElement* FindLogicalElement(ElementPath* 
elementpath); 

In the above API implementation, an ElementPath type parameter related to a target UI 
element is received from a client program (e.g., 720 of FIG. 7), a search is performed 
accordingly and a result of LogicalElement type or any local indicator of the target UI 
element in target program is retumed. 

A find method can also be implemented that receives a string type description of 
an element path as an input parameter. Such an implementation may be as follows: 

Public: static LogicalElement* FindLogicalElementString(String* 
elementpathstring); 

In this API the input parameter is a string type description of the element path and it 
complements the GetElementPathString(LogicalElement * logical element) API 

These APIs may be further refined by adding other parameters. For instance, 
GetElementPath API can have a flag parameter that could be set such that sibling order 
of each UI element in an element path hierarchy is recorded as part of the ElementPath 
result when the flag is set to be true. Then later in the FindLogicalElement the sibling 
order of UI elements in an element path may be useful for resolving ambiguity. The 
same may apply to the GetElementPathString and FindLogicalElementString APIs 
Other specific implementations of the APIs are possible that implement the 
fiinctionahty described above without the exact same naming convention for the 
methods and class names. 
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Other improvements are possible. For instance, if the element path root node 
need not be the topmost node in a hierarchy (e.g., 215A) then identifier information of 
whatever is the element path root node may be provided as a parameter of the 
GetElementPath API so that only information regarding the UI element of interest and 
5 the element path root node are recorded. Also, FindLogicalElement can have as a 
parameter the element path root node so that the search from a UI element is begun at 
the designated element path root. 

Other specific implementations of the APIs are possible that implement the 
functionahty described above without the exact same naming convention for the 
10 methods and class names. In fact, the implementations need not even be in an object- 
oriented language. In fact, the implementations need not even be in an object-oriented 
language. 

Alternatives 

15 

Having described and illustrated the principles of our invention with reference to 
the described embodiments, it will be recognized that the described embodiments can 
be modified in arrangement and detail without departing from such principles. 
Although, the technologies described herein have been illustrated via examples of 

20 elements of graphical user interfaces, any of the technologies can use other user 

interface elements (e.g., sound). The APIs described herein have been illustrated with 
selected input parameters for the purposes of describing functionality of generating 
identifiers and searching by identifiers. Other parameters may be added or some of the 
parameters may be removed and still implement the principles of the invention. The 

25 same appHes to the output parameters. Also, the element path identifiers have been 
described using selected component identifiers of elements in an element path. Other 
identifiers as appropriate for a selected user interface environment may be used as well. 
Furthermore, the functionality described herein with respect to the generation of 
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identifiers and their use in searching and locating interface elements may be 
implemented across multiple software component modules. For instance, another 
software component may be introduced between the API modules and the client 
program or the element path engine to pass fimction calls, receive results and conduct 
5 other comparable actions. 

Also, it should be understood that the programs, processes, or methods described 
herein are not related or limited to any particular type of computer apparatus. Various 
types of general purpose or specialized computer apparatus may be used with or 
perform operations in accordance with the teachings described herein. Actions 

10 described herein can be achieved by computer-readable media comprising computer- 
executable instructions for performing such actions. Elements of the illustrated 
embodiment shown in software may be implemented in hardware and vice versa. In 
view of the many possible embodiments to which the principles of our invention may be 
applied, it should be recognized that the detailed embodiments are illustrative only and 

15 should not be taken as limiting the scope of our invention. Rather, we claim as our 
invention all such embodiments as may come within the scope and spirit of the 
following claims and equivalents thereto. 



